IMPROVING  DOD  MAINTENANCE  THROUGH  BETTER 
PERFORMANCE  AIDS 

Thomas  C.  Rowan 

Information  Concepts,  Incorporated 


National  Technical  Information  Service 
U.  S.  DEPARTMENT  OF  COMMERCE 
5285  Port  Royal  Road,  Springfield  Va.  22151 


AD  758713 


IMPROVING  DOD  MAINTENANCE 
THROUGH  BETTER  PERFORMANCE  AIDS 


Final  Report  -  Article  1 -d 
1  March  1973 


by 

Thomas  C.  Rowan 
Project  Scientist 


Sponsored  by: 

Advanced  Research  Projects  Agency 
ARPA  Order  No.  2244 
Program  Code  No.  3D20 


This  research  was  supported  by  the  Advanced 
Research  Projects  Agency  of  the  Department  of 
Defense  under  Contract  No.  DAHC  15  70  C  0257 


Contract  Expires  30  April  1973 


Reproduced!  by 

NATIONAL  TECHNICAL 
INFORMATION  SERVICE 


U  S  DeparlmcM  of  Commerce 
Springfield  VA  27151 


EEEH1IEI 


INFORMATION  CONCEPTS  INCORPORATED 


I 


UNCLASSIFIED 


1. 1 


uul\  CliisMftt  »ln*n 


DOCUMENT  CONTROL  DATA  •  R  2.  D 

|  unli  «  AfiMft  it/  tit! *#/  ithstrm  t  *tiu!  «»#*  no  f/if  ton  iiti*  ••  hr  ettletrtl  w/iwi  //ir*  <»y<  n$ll  /rport  K  1 1/f.slHrt!) 

\  OHioiti  A  tiMG  AC  II  vi  Tv  (l^nrpnrnt*  mithttt) 

rcfoii  t  sKcuiiirv  clav^kicaiic u 

UNCLASSIFIED 

Information  Concepts  Incorporated 

2h.  CROUP 

J  RCPORT  TITLE 


improving  DOD  Maintenance  Through  Better  Performance  Aids 


4.  DESCRIPTIVE  NOTES  (Type  ot  repot t  end.inctatlve  dates) 

Final  Report  -  Article  1-d,  1  March  1973 

6.  AO  TMORlSl  (First  n*rnct  middle  initial ,  last  nemo) 


Thomas  C.  Rowan 


&.  REPORT  DATE 

March  1,  1973 

7«.  TOTAL  NO.  O^PAGES 

_ KlL _ 

7b.  NO.  OF  REFS 

26 

fa.  CONTRACT  OR  GRANT  NO. 

04.  ORIGINATOR'S  REPORT  NUMOCrt(S)  1 

DAHC  15  70  C  0257 

6.  PROJECT  NO. 

None 

C. 

Ob.  other  REPORT  UOI5}  (Arty  ether  numbora  that  may  be  aeetQned 

this  report) 

ARPA  Order  No.  2244 

d. 

tO.  GISTRI  OUT  ION  STATEMCNt 

Distribution  Unlimited 

_ 

11.  SUPPLEMENTARY  notes 

12.  SPONSORING  MILITARY  ACTIVITY 

None 

Advanced  Research 

Projects  Agency 

13.  AOSTRACT 


This  report  is  concerned  with  the  technical  documentation  used  by 
maintenance  personnel,  and  in  particular  job  performrnce  'aids,  that  is 
documents  or  devices  that  give  precise  step  by  step  directions  to 
perform  a  task  or  otherwise  furnish  the  technician  all  the  relevant 
information  he  needs  in  a  compact  form.  A  review  is  made  of  past 
research  and  current  projects  in  DOD  and  elsewhere  on  maintenance 
aiding  through  innovations  in  format  and  content  of  maintenance 
documentation.  Problems  in  evaluating  this  research  and  in  imple¬ 
menting  proven  techniques  are  discussed.  JXfecommenda.tions  are  made* 
with  respect  to  additional  research  and  system  demonstrations 
needed,  areas  for  implementation  and  potential  savings  in  changing  v 
training  and  personnel  assignment  practices.  * 


FORM 

t  NOV  6* 


1473 


VN  0102-014.6700 


(PAGE  1} 


UNCLASSIFIED 

Security  Classification 


Maintenance 
Maintenance  Training 
Maintenance  Documentation 
Job  Performance  Aids 


dd  :t..u 

S/N  OIOt*$Q7*«82t 


1473  fBACK> 


UNCLASSIFIED 

Security  Classification 


The  views  and  conclusions  contained  in  this 
document  are  those  of  the  author  and  should 
not  be  interpreted  as  necessarily  repre¬ 
senting  the  official  policies,  either 
expressed  or  implied,  of  the  Advanced  Research 
Project  Agency  or  the  U.S.  Government. 


THOMAS  C.  ROWAN 
Project  Scientist 


IMPROVING  DOD  MAINTENANCE 
THROUGH  BETTER  PERFORMANCE  AIDS 


TABLE  OF  CONTENTS 

Page 

SUMMARY  .  1 

PERFORMANCE  AID  TECHNIQUES . .  .  .  .  .  8 

REVIEW  OF  PERFORMANCE  AID  RESEARCH .  15 

Experiments  and  Field  Tests  .  ....  15 

Other  DO D  Projects . . .  32 

Non-DOD  Developments  .  36 

DISCUSSION  OF  PERFORMANCE  AIDS .  39 

Difficulty  of  Comparative  Studies  .  39 

Institutional  Change  Problem  .  ....  44 

Personnel  and  Training  Implications  .  44 

Cost  Savings  Potential . 47 

Task  Analysis  .  .  ..  . . 50 

Need  for  Definitive  Experiment  .........  52 

INFORMATION  PRESENTATION  MEDIA  .  56 

RECOMMENDATIONS  . .  59 

REFEuSTCES . 64 


ED 


INFORMATION  CONCEPTS  INCORPORATED 


If 


i 


J- 

SUMMARY 

DOD  spends  on  the  order  of  $20  B  annually  for  maintenance r 
of  which  more  than  half  is  for  personnel.  There  are  indi¬ 
cations  that  there  is  room  for  major  improvement  in  the  per¬ 
formance  of  maintenance  with  potential  for  reducing  costs  and 
increasing  equipment  availability.  Such  indications  include 
high  removal  rates  of  non-failed  components  (on  the  order  of 
30%)  and  large  disparities  in  the  costs  of  maintaining 
similar  equipment  by  different  organizations.  Although  con¬ 
siderable  pressure  has  developed  in  the  hardware  area  for 
improvement  in  maintenance  including  stressing  reliability 
and  maintainability  as  design  criteria  and  developing  auto¬ 
matic  and  built  in  test  equipment,  little  attention  has  been 
given  to  improving  information  transfer  to  the  technician  on 
the  job. 

Numerous  studies  on  the  format  and  content  of  maintenance 
information  indicate  that  faster  and  better  maintenance  can  be 
done  if  technicians  use  job  performance  aids,  that  is,  docu¬ 
ments  or  devices  that  give  precise  step-by-step  instructions 
for  each  task  or  otherwise  present  in  a  concise  and  consolidated 
manner  all  information  relevant  to  that  task. 

DOD  has  spent  less  than  $5  M  to  date  and  is  spending 
considerably  less  than  $1  M  annually  to  improve  this  "people" 
part  of  maintenance.  The  low  level  of  R&D  that  has  been 
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accomplished  over  the  past  several  years  indicates  a  large 
potential  for  improving  maintenance  and  reducing  both 
maintenance  and  training  costs.  More  specifically  the  results 
indicate  a  potential  to: 

a)  Reduce  repair  time  and  false  removals  and  improve 
equipment  reliability. 

b)  Significantly  reduce  (up  to  75%)  the  training 
required  for  inexperienced  personnel  to  perform  accurately 
routine  maintenance  tasks. 

c)  Productively  utilize  personnel  of  lower  ability 
levels . 

d)  Achieve  the  above  at  costs  for  maintenance  aids 
comparable  (within  20  -  25%)  to  those  now  being  expended  for 
conventional  technical  documentation. 

Problems  within  DOD  which  must  be  solved  for  the  potential 
improvement  and  savings  to  be  realized  include: 

a)  Lack  of  visibility  and  focus  on  the  costs  of  improper 
maintenance . 

b)  Institutional  resistance  to  change  and  a  lack  of  a 
sense  of  urgency  to  try  new  approaches  in  maintenance  and  in 
maintenance  training. 

c)  Current  fragmented  and  inadequately  supported  R&D 
efforts  to  demonstrate  the  improvement  potential. 

d)  Lack  of  focal  points  in  OSD  and  the  military  depart¬ 
ments  responsible  for  innovating  and  measuring  improvement  in 
the  field. 
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In  order  to  address  these  problems:' 

a)  OSD  and  the  military  departments  must  establish  focal 
points  with  a  commitment  to  achieve  a  technology  transfer  from 
the  R&D  level  to  tho  field  and  to  measure  the  results. 

b)  The  R&D  must  be  adequately  supported,  perhaps  at  a 
level  of  $4-5  M  per  year  across  DOD.  The  principal  funding 
requirement  is  to  plan,  conduct  and  measure  the  results  of 
several  large  scale  field  demonstrations  of  sufficient  scope 
and  time  duration  to  resolve  issues  of  cost,  environment  and 
attitude. 

c)  There  must  be  a  commitment  at  the  top  levels  of  OSD 
and  the  military  departments  to  support  major  demonstration 
programs  which  cross  the  lines  of  systems,  logistics  and 
training  and  in  areas  which  have  a  natural  institutional 
•resistance  to  change. 

More  specifically,  the  following  summary  recommendations 
are  given  below  based  on  material  developed  in  the  body  of  the 
report : 

\ 

System  Demonstration 


There  is  a  need  for  a  program  of  definitive  demonstrations 
to  settle  the  main  issues  in  the  area  and  to  provide  the 
needed  impetus  fcr  implementation  of  those  features  that  are 
effective  in  a  real  operating  environment.  The  needed  demon¬ 
strations  should  be  conducted  with  each  of  the  military 
departments  being  involved. 
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(1)  The  proposed  but  unfunded  Air  Force  project 
"Innovate"  should  be  enlarged  and  carried  out.  This  project 
will  compare  conventional  technical  orders ,  decision  aids 
and  fully  proceduralirsed  job  performance  aids  (JPA's)  and 
will  provide  answers  to  the  questions  of  whether  brief 
technical  training  is  adequate  with  JPA's  and  whether  JPA's 
alleviate  the  cross-training  problems  within  a  specialty. 

(2)  The  Army's  "Low  Cost  Ownership"  Program  on  the 
improvement  of  military  technical  information  transfer  methods 
should  be  supported  and  funded  as  a  three  year  experiment. 

This  effort  will  analyze  information  presentation  approaches  ' 
in  terms  of  their  application  to  commodity  groups  -  aircraft, 
combat  vehicles,  etc.  -  and  will  select  the  best  approach  or 
combination  of  approaches  for  each  commodity  group.  Packages 
will  then  be  demonstrated  for  several  military  occupational 
specialties  for  maintenance  up  through  the  depot  level. 

(3)  The  Naval  Air  Systems  Command's  MIARS  and  TRUMP 
systems  should  be  accelerated  and  considered  for  Navy-wide 
use.  MIARS  is  a  microfilm  system  with  hard  copy  compatibility 
that  involves  the  reorganization  of  technical  data  into  "work 
packages"  that  cut  down  the  time  that  technicians  must  spend 

•k 

searching  through  manuals.  It  offers  the  capability  of  being 
the  vehicle  for  introducing  improvements  in  content  such  as  job 
performance  aids,  once  these  improvements  are  accepted.  TRUMP 
is  a  system  now  in  procurement  for  automatically  updating  MIARS. 
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Management: 

(1)  Focal  points  should  be  designated  in  OSD  and  in  each 
of  the  military  departments  with  responsibility  to  deal  with 
the  R&D,  systems,  logistics,  and  training  efforts  needed  for 
the  large  scale  demonstration  program. 

(2)  There  should  be  fiscal  visibility  and  management 
focus  for  the  R&D  program  for  maintenance  aids.  A  Program 
Element  should  be  established  for  each  military  department  to 
consolidate  the  efforts. 

(3)  In  order  to  coordinate  the  overall  DOD  efforts, 
there  should  be  a  tri-service  review  of  current  activities 
and  plans,  in  the  maintenance  aid  area.  A  committee  should  be 
formed  or  an  existing  one  directed  to  review  the  ongoing 
activity,  particularly  the  planning  of  major  demonstrations 
and  the  utilization  of  the  results  of  those  demonstrations. 
Formulation  of  an  ARPA  effort  similar  to  the  CAI  program 
initiative  should  be  considered. 

(4)  For  an  interim  period  of  two  to  three  years,  OSD, 
specifically  OSD  (I&L)  and  DDR&E,  should  review  the  maintenance 
aids  approach  and  funding  proposed  for  new  weapon  systems 

and  should  stimulate  the  application  of  innovative  approaches 
and  funding  support  for  these. 

(5)  The  military  departments  should  initiate  analyses 
on  the .economics  of  production  and  utilization  of  improved 
maintenance  aids.  These  should  include  the  comparative  costs 
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of  alternative  format  and  content  (e.g.,  JPA's,  SIWMS,  con¬ 
ventional  manuals)  as  veil  as  media  (books,  microfilm,  video 
cassette) . 

(6)  DOD  should  institute,  through  the  military  depart¬ 
ments,  an  analysis  of  the  costs  of  improper  maintenance  (to 
include  such  factors  as  fake  removals,  spares,  shipping,  etc.)* 
This  data  should  be  included  in  a  broad  analysis  of  the 
economics  of  applying  better  maintenance  aids  and  information 
systems  -  to  include  training  costs,  personnel  costs,  and  ■ 
equipment  costs. 

Systems  Application: 

(1)  Non-troubleshooting  fully  proceduraiized  job  per¬ 
formance  aids  —  job  guides  —  should  be  developed  for  all 
new  weapons  systems.  These  will  be  primarily  oriented  to 
routine  maintenance  tasks.  This  should  be  done  in  accordance 
with  specifications  requiring  task  analysis. 

(2)  Job  guides  should  be  developed  for  selected  portions 
of  systems  now  in  the  inventory,  based  on  a  review  of  high 
cost  areas. 

(3)  Information  systems  other  than  books  such  as  audio¬ 
visual  cassette  and  film  systems  to  provide  system  overviews, 
to  introduce  equipment  modifications  and  to  explain  difficult 
procedures  should  be  applied. 

(4)  For  major  weapon  systems,  microfilm  reader  printers 
which  include  job  guide  formatted  materials  should  be  used 
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both  for  economic  update  and  facility  of  information 
access. 

Research  and  Development: 

(1)  FEFI/TAFI  (Flight  Environment  Fault  Indication/ 

Turn  Around  Fault  Isolation) ,  a  troubleshooting  system  for 
the  DC-10,  should  be  examined  for  possible  military  appli¬ 
cation. 

(2)  Aids  useful  at  different  skill  and  experience  levels 
and  aids  that  have  trai.ning  objectives  should  be  developed. 

The  promising  Navy  sponsored  research  on  "families  of  aids" 
should  be  extended  so  that  multipurpose  documentation 
suitable  for  experienced  as  well  as  inexperienced  tech¬ 
nicians  can  be  developed  with  consideration  also  given 

to  the  inclusion  of  OJT  material. 

(3)  Computer  supported  audio-visual  presentation 
systems  should  be  studied  as  potential  sophisticated  trouble¬ 
shooting  aids  making  possible  dynamically  optimized  trouble¬ 
shooting  strategies. 

(4)  Studies  should  be  made  of  the  influence  that  system 
characteristics  such  as  modularity  or  degree  of  built-in 
testing  have  on  the  type  of  aids  appropriate  for  a  system. 
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PERFORMANCE  AID  TECHNIQUES 


Maintenance  accounts  for  a  large  portion  (20-25%)  of 
the  total  DOD  budget  and  even  a  small  percentage  decrease 
in  this  area  would  represent  substantial  savings.  Over  the 
past  ten  years  DOD  has  put  much  emphasis  or  hardware  approaches 
to  better  maintenance  including  more  attention  to  reliability 
and  maintainability  as  design  criteria  and  the  development 
of  automatic  and  built-in  test  equipment. 

One  important  area  has,  however,  received  very  little- 
attention  and  that  is  the  area  of  maintenance  information. 

The  conventional  technical  manual  systems  have  existed  with¬ 
out  major  change  for  decades.  Some  gradual  improvement  has 
occurred  but  there  has  been  no  innovative  thrust  to  do  some¬ 
thing  significantly  better  for  the  "people"  part  of  the 
maintenance  problem.  Within  the  R&D  community  there  have  been 
a  number  of  small-scale  experiments  and  demonstrations  of 
new  ways  of  presenting  information  that  enable  technicians 
to  perform  faster  and  more  accurately,  and  with  less  training. 
Although  some  of  these  techniques  appear  to  have  the  potential 
for  greatly  reducing  cost  of  maintenance  perhaps  by  30%  or 
more  and  improving  equipment  availability,  there  has  been 
relatively  little  application  of  these  research  results  out¬ 
side  of  the  laboratory.  This  report  discusses  what  has  been 
done  in  developing  alternatives  to  the  conventional  technical 
manual  and  recommends  several  steps  that  DOD  should  take  in 
the  area . 
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The.  report  is  concerned  primarily  with  what  have 
become  known  as  performance  aidsf  that  is,  devices  or  docu¬ 
ments  that  contain  guidance  information  that  helps  the 
technician  perform  a  particular  job  at  hand,  and  are  "people*1 
rather  than  equipment  directed.  The  aid  may  be  a  very  specific 
step-by-step  set  of  directions  that  require  no  decisions  to 
be  made  cr  it  may  be  something  that  gives  a  more  general 
picture  of  the  system  and  assists  the  technician  in  deciding 
what  to  do  next.  A  large  number  of  approaches,  and  acronyms, 
have  arisen  but  most  of  the  research  has  tended  to  focus  on 
the  specific  or  fully  proceduralized  aid  for  non-trouble¬ 
shooting  tasks  with  the  differences  in  approach  appearing  in 
the  area  of  troubleshooting  aids. 

The  objective  of  all  the  approaches  has  been  to  provide 
in  one  package  simple,  complete  and  current  information  to 
the  technician,  without  the  need  for  cross  referencing  and 
retention.  Fully  proceduralized  performance  aids  break  up 
a  task  into  easily  remembered  steps  and  present  unambiguous 
directions  using  simple  English  and  relevant  illustrations. 

Well  designed  aids  are  based  on  a  careful  analysis  of  the 
task  the  technician  actually  has  to  do  and  take  into  con¬ 
sideration  the  amount  of  training  and  experience  of  the 
personnel  who  will  be  using  them.  While  one  would  assume 
that  \.he  current  maintenance  documentation  would  do  just  this, 
it  rarely  does.  The  process  by  which  documentation  is  produced 
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often  means  that  the  maintenance  manual  is  written  before 
the  equipment  is  produced  and  is  not  based  on  what  the 
technician  actually  must  do  to  repair  it.  In  analyzing 
maintenance  actions  required  for  the  doppler  radar  system  of 
the  C-141,  the  Air  Porce  found  that  the  isolation  and  repair 
of  one  malfunction  required  reference  to  165  pages  of  8 
documents.  If  no  false  moves  were  made,  41  changes  in  docu¬ 
ment  location  were  required. 

Fully  procedural i zed  aids  have  been  developed  for  trouble¬ 
shooting  tasks,  but  most  of  the  research  on  troubleshooting 
has  involved  decision  aids,  devices  for  helping  the  technician 
understand  the  system  or  subsystem  and  formulate  a  strategy 
for  isolating  the  fault  or  faults.  Most  of  the  decision 
aids  have  included  variations  of  the  Maintenance  Dependency 
Chart  (MDC) ,  a  device  which  can  vary  greatly  in  complexity 
and  which  indicates,  for  different  system  modes,  what  elements 
are  interrelated. 

Jobs  performance  aids  research  has  been  summarized  a 
number  of  times;  one  of  the  most  recent  summaries  is  in  Price 
et  al  (3.S71) .  Others  are  Foley  (1969) ,  Chalupsky  and  Kopf 
(1967)  and  Shriver  (1966) .  For  the  most  part  these  have 
been  descriptive  summaries  rather  than  critical  reviews.  An 
attempt  will  be  made  here  to  evaluate  the  quality  of  selected 
research  projects  in  order  to  assess  the  present  claims  put 
forward  for  this  general  approach  to  maintenance  documentation 
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and  for  particular  aids.  All  known  reports  that  included 
laboratory  or  field  test  results  will  be  reviewed;  in  the 
interests  of  brevity  only  the  highlights  of  each  study  will 
be  covered.  More  detail  can  be  found  in  the  specific 
references  given  or  in  the  above  mentioned  reviews.  The 
studies  are  summarized  in  Table  1. 
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REVIEW  OF  PERFORMANCE  AID  RESEARCH 


EXPERIMENTS  AND  FIELD  TESTS 

FORECAST,  which  resulted  from  early  HumRRO  research, 
is  an  approach  to  systems  analysis,  training  and  performance 
aids.  The  aids  produced  are  decision  type  aids  such  as 
troubleshooting  block  diagrams,  waveform  guides,  and  blocked 
schematics.  These  aids  are  used  in  the  training  program  to 
convey  system  understanding  and  to  teach  troubleshooting. 

In  1958,  (Shriver,  1960  -  Report  #63)  an  experimental  12  week 
FORECAST  training  program  was  conducted  for  Army  M-33  anti¬ 
aircraft  fire  control  system  electronic  technicians.  A 
comparison  of  20  experimental  subjects  and  17  students  from 
the  regular  30  week  course  was  made  using  89  troubleshooting 
tasks  administered  over  a  nine  day  period.  The  mean  test 
score  for  the  experimental  group  was  140.8  and  the  standard 
group  142.9.  With  standard  deviations  of  9.2  for  the 
experimental  group  and  11.7  for  the  standard  group,  the 
difference  can  be  considered  slight.  Of  importance,  of 
course,  is  the  difference  in  training  time  -  12  weeks  versus 
the  standard  30  weeks.  The  relative  contribution  to  per¬ 
formance  of  tne  training  program  versus  the  troubleshooting 
aids  cannot  be  deduced.  In  a  later  study  students  who  were 
given  additional  practice  with  a  mock-up  of  the  real  equipment 
performed  significantly  better  than  the  earlier  groups.  In 
this  case,  the  effects  of  the  experimental  training,  the 
performance  aids  and  the  use  of  the  mock-up  are  confounded. 
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Another  test  was  conducted  in  1963  with  98  subjects 
using  Navy  Loran  equipment.  Twelve  subjects  had  conventional 
training  and  manuals;  the  remaining  had  FORECAST  training  and 
manuals.  The  FORECAST  subjects  identified  three  times  as 
many  malfunctions  as  the  conventional  subjects. 

JOBTRAIN  was  also  the  result  of  research  by  HumRRO . 

Under  this  concept  a  comprehensive  failure-effect  analysis 
is  made  and  symptom  patterns  developed.  The  resulting  job' 
aid  is  in  paragraph  form  with  branching  to  subtests  when  bad 
effects  are  found.  An  experimental  training  program  was 
developed  to  be  used  in  conjunction  with  the  aid.  The  aid 
and  training  course  were  compared  with  traditional  documen¬ 
tation  and  training  in  1962  at  Fort  Gordon,  Georgia,  using 
radio  relay  and  carrier  repair  technicians  (GEBHARD,  1970) . 
Nineteen  standard  trainees  (regular  25  week  school,  con¬ 
ventional  documents)  and  twenty-one  experimental  trainees 
(JGBTRAIN  course  of  11  .weeks  and  the  JOBTRAIN  aid)  were 
given  18  problems  to  troubleshoot  and  were  also  tested  on 
their  ability  to  align,  remove,  replace,  etc.  Variables 
such  as  education,  electronic  aptitude  and  civilian 
experience  were  controlled  by  matching  subjects.  The 
results  of  the  tests  were  that  the  groups  performed  similarly. 
No  statistically  significant  differences  were  found  and  the 
investigators  concluded  therefore  that  JOBTRAIN  training  and 
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aids  were  as  effective  as  traditional  training  and  manuals 
and  that  a  50%  reduction  in  training  time  can  result  from 
the  JOB*? RAIN  approach.  A  problem  with  such  a  conclusion  is 
that  in  terms  of  statistical  theory,  failure  to  find  a 
difference  does  not  prove  that  no  difference  exists. 
Practically  speaking,  however,  the  results  are  suggestive. 

One  is  left  with  uncertainty  about  the  relative  importance 
of  the  experimental  training  course  versus  the  job  aid. 

MAINTRAIN,  (Rogers  &  Thomas,  1963)  another  HumRRO 
project,  was  an  effort  to  develop  an  improved  manual  for  use 
by  trained  technicians  in  troubleshooting  complex  electronic 
equipment.  Included  in  the  manual  were  an  improved  index  to 
troubleshooting  instructions,  data  flow  diagrams,  and  chassis 
function  diagrams.  The  manual  was  tested  by  having  recent 
Missile  School  graduates  troubleshoot  44  faults  with  Nike 
Ajax  Missiles.  Subjects  were  matched  on  the  basis  of  Air 
Defense  School  grades  and  divided  into  two  groups  of  eight. 

One  group  used  the  MAINTRAIN  Manual  and  the  other  conventional 
manuals.  The  experimental  group  located  42%  more  electronic 
malfunctions  in  41%  less  time.  These  differences  are 
significant  at  the  .10  level  of  confidence.  Why  such  a 
confidence  level  was  used  is  not  indicated.  The  number  of 
electronical  faults  found  by  the  two  groups  was  not 
statistically  different. 
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In  1964-65  HumRRO  developed  a  procedure  oriented  training 
program  for  HAWK  Radar  Mechanics  (HumRRO  1969) .  As  part  of 
this  project,  symptom-collection  manuals  were  developed  which 
were  essentially  procedural! zed  troubleshooting  aids.  The 
manuals  were  used  in  training  a  30  man  experimental  group. 

In  a  test,  experimental  students  using  the  symptom-collection 
manuals  (and  traditional  documentation)  isolated  80%  of  the 
defective  chassis  while  the  control  group  using  only  traditional 
documentation  isolated  40%.  This  difference  is  statistically 
significant.  Further  analysis  indicated  that  the  utility  of 
the  symptom-collection  manuals  was  greatest  for  the  most 
difficult  malfunctions.  The  experimental  and  control  groups 
were  quite  comparable  in  age  and  education.  Their  aptitude 
scores,  however,  differed;  the  average  general  technical 
aptitude  (GT)  score  for  the  experimental  group  was  123.4. 

The  averages  for  the  two  control  groups  were  115.8  and  118.5. 
Whether  this  accounts  for  the  difference  in  performance  is 
unknown,  but  the  possibility  weakens  the  impact  of  the  study. 

The  SAFZ3GUARD  Maintenance  Data  System  (MDS) ,  to  be 
discussed  later,  derived  from  earlier  work  with  the  NIKE  X 
system  performed  by  Bell  Telephone  Laboratories.  In  1964, 
a  data  system  called  A-VIS,  (Audio  -  Visual  Information 
System)'  was  tested  at  White  Sands  Missile  Range  (USATECOM, 

1964) .  As  the  name  implies  this  is  an  audio  visual  system 
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and  the  test  consisted  of  comparisons  of  modes  of  presenting 
maintenance  information.  The  content  used  consisted  of 
both  logic-tree  trouble  analysis  aids  and  decisions  aids  such 
as  functional  block  diagrams,.  This  JPA-like  content  was 
presented  in  a  combined  audio-visual  mode,  an  audio  mode 
(with  the  visual  portion  in  hard  copy) ,  a  visual  mode  (audio 
portion  in  hard  copy)  and  a  hard  copy  mode,  termed  programmed 
TM.  All  of  these  modes  of  presentation  were  superior  to 
technical  manuals  (acutaliy  TM-like  material  since  no  TM's- 
existed  for  this  R&D  equipment) .  Of  the  presentation  modes 
the  visual-only  one  was  superior.  In  terms  of  a  performance 
index  created  by  the  investigators,  the  visual-only  mode  was 
27%  more  effective  than  technical  manuals.  The  programmed 
TM  was  19%  more  effective. 

Shriver  (1966)  reports  in  his  review  an  evaluation  of 
SIMMS  that  was  carried  out  for  the  Coast  Guard  in  1964  on  a 

radar  with  42  subjects.  The  SIMMS  group  identified  the 

\ 

three  malfunctions  used  in  the  test  96%  of  the  time  and  the 
conventional  group  identifi.ed  them  70%  of  the  time.  The 
SIMMS  group  took  half  as  much  time  to  troubleshoot. 

Research  Triangle  Institute  (Krohn,  et  al,  1971)  under 
contract  to  Naval  Ships  Engineering  Center  compared  SIMMS 
and  conventional  documentation  for  the  AN/SRC-20  radio  set 
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at  three  Navy  C-school  locations  involving  178  technicians. 
Each  school  had  classes  on  the  SRC-20,  some  based  on  the 
conventional  manual  and  some  on  the  SIMMS  manual.  Identical 
instructional  material,  tests  and  troubles  were  used  in  each 
school.  Data  on  troubleshooting  performance  were  collected 
for  one  year  by  instructors.  Analysis  indicated  that  data 
from  the  different  schools  could  not  be  combined  because 
different  standards  were  used  by  the  instructors  in  collecting 
the  data.  Because  of  the  fact  that  the  students  we re  not  . 
matched  for  relevant  characteristics  when  assigned  to  the 
two  versions  of  the  course,  the  results  were  hard  to  inter¬ 
pret.  The  investigators  concluded  that  technicians  with 
medium  training  performed  similarly  with  either  manual. 

Those  with  little  training  did  better  with  SIMMS  and  highly 
trained  technicians  did  better  with  conventional  documen¬ 
tation.  In  the  discussion,  the  investigators  state  that 
other  factors  such  as  training  and  experience  may  have 
biased  the  results.  For  example,  in  the  low  experience 
SIMMS  group,  which  did  better  than  the  corresponding  con¬ 
ventional  group,  16  of  the  28  technicians  had  prior  experience 
with  the  SRC-20  as  compared  to  4  of  the  14  conventional 
technicians.  This  and  other  weaknesses  in  the  study  make  the 
results  inconclusive. 


Binary  Fault  Isolation  Chart  (BFIC) ,  a  concept  developed 
by  Westinghouse,  is  a  troubleshooting  logic-tree  using  a 
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flow  chart  format  with  standard  geometric  symbols.  An 
experiment  was  conducted  in  1966  with  80  Westinghcuse 
employees,  matched  in  experience  and  aptitude  {.Westing- 
house,  1966) .  Half  the  group  was  given  a  BFIC  for  use  in 
troubleshooting  a  moderately  complex  electronic  module. 

The  others  were  given  traditional  documentation.  The 
experimental  group  performed  in  one-third  the  time  taken 
by  the  control  group  and  made  one-fifth  the  errors.  The 
results  were  significant  at  the  .01  level  of  confidence.  ■ 

Project  PIMO  (Presentation  of  Information  for  Main¬ 
tenance  and  Operations)  has  been  the  most  extensive  effort 
in  the  job  aid  area.  It  was  carried  out  by  Serendipity 
Inc.  for  the  Air  Force  from  1964  to  1969  and  is  described 
in  an  eight  volume  final  report  (Serendipity,  1969) .  PIMO 
consists  of  fully  proceduralized  aids  for  non-troubleshooting 
maintenance  tasks  (job  guides)  and  simplified  maintenance 
dependency  charts  for  troubleshooting.  The  guides  used  a 
fixed  format  and  syntax,  and  a  preferred  verb  list,  with  a 
limited  number  of  steps  per  page.  The  concept  is  often 
associated  with  format  -  pocket  size  books  with  pictures 
and  text  on  facing  pages  -  but  like  the  other  aiding 
concepts,  its  content  necessarily  must  be  based  on  careful 
task  analysis. 
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A  large  scale  field  evaluation  was  carried  out  primarily 
at  Charleston  AF  Base  in  1968  using  aids  produced  for  organi¬ 
zational  level  maintenance  or  the  C-141A.  PIMO  booklets, 
an  audio  visual  version  of  PIMO  and  traditional  documentation 
were  compared  using  both  exnerienced  and  inexperienced 
personnel.  Perhaps  the  most  noteworthy  result  of  the  evalu¬ 
ation  was  that  in  over  1,000  observations  of  performance  with 
PIMO  no  error  was  observed. 

In  non-troubleshooting  activities,  experienced  personnel 
using  standard  operating  procedures  had  shorter  maintenance 
times  than  they  did  when  using  PIMO.  Analysis  of  the  data 
indicated,  however,  that  time  to  repair  using  PIMO  was 
reduced  with  repetition  and  approached  the  conventional  time. 

Apprentice  technicians  performed  non-troubleshocting 
tasks  without  error  using  PIMO  in  slightly  more  time  than  the 
experienced  technicians  did  when  they  used  PIMO.  The  PIMO 
report  authors  point  to  the  practical  significance  of  the 
apprentices  being  able  to. perform  maintenance  error  free  using 
PIMO  even  though  they  were  slightly  slower.  It  was  assumed 
that  their  times  would  also  be  reduced  with  repetition  as 
they  learned  the  equipment  and  the  aids. 

One  of  the  objectives  of  the  field  test  was  to  compare 
the  audio-visual  PIMO  mode  with  the  hard  copy  mode.  The 
results  were  that  performance  times  using  the  booklet  were 
slightly  shorter  but  the  difference  was  statistically 
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significant  only  for  Remove  ‘Activities .  The  authors  con¬ 
cluded  that  the  modes  were  equivalent  and  thus  booklets, 
being  less  .expensive,  were  preferable.  It  is  worth  noting 
that  considerable  difficulty  was  experienced  during  the 
test  with  the  audio-visual  hardware  which  was  reflected  in 
the  opinions  expressed  by  participants.  What  impact  this 
had  on  results  is  unknown . 

Only  experienced  personnel  were  used  in  testing  the 
PIMO  troubleshooting  aids  (TSA) .  There  was  an  11%  reduction 
in  time  using  PIMO  versus  standard  procedures?  this  is 
statistically  significant.  Forty  troubleshooting  actions 
were  performed  with  PIMO  with  one  error,  a  zero-defect 
removal.  Forty-one  actions  were  taken  in  the  standard 
mode  with  12  errors;  five  of  these  were  zero-defect  removals. 
These  differences  were  not  tested  for  statistical  signifi¬ 
cance,  but  are  interesting  considering  the  impact  a  reduction 
of  5  to  1  in  zero-defect  removals  would  have  in  terms  of 
spares  consumption. 

Applied  Science  Associates,  under  contract  to  AFHRL 
in  1968  compared  20  naive  high  school  students  given  12  hours 
training  with  41  Air  Force  technicians  having  several  years 
training  and  experience  in  electronics  (Elliott  &  Joyce, 

1968).  .  The  average  technician  was  27  years  old,  had  attended 
three  Air  Force  schools  and  was  between  grade  E-4  and  E-5.  The 
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me»n  AQE  electronic  aptitude  score  was  83  for  the  technicians, 
73  for  the  high  school  students.  The  task  was  within-stage 
troubleshooting  on  the  Maintenance  Task  Simulator  (KTS-2) . 

The  students  used  fully  proceduralized  aids  and  the  technicians 
tech-order  like  data.  Unfortunately  the  tasks  given  the  two 
groups  differed.  The  students  were  given  only  one  chance  to 
solve  each  problem  while  the  technicians  were  given  a  time 
limit  of  90  minutes.  The  student  mean  time  to  completion  was 
thus  the  average  of  the  times  taken  to  complete  the-  specified 
procedure  once.  Some  of  the  technicians  took  the  full  90 
minutes,  thus  greatly  influencing  their  mean  time  with  the 
result  that  it  (32.7  minutes)  was  greater  than  the  students' 
(27.5  minutes).  On  the  other  hand  the  students  made  2-1/2 
times  more  errors:  if  the  module  did  not  function  properly 
at  the  end  of  the  checkout  procedure,  the  student  wasn't 
allowed  to  go  back  to  find  where  he  erred,  whereas  the 
technicians  were  limited  only  by  time. 

The  investigators  tried  to  estimate  what  the  results 
would  have  been  if  the  students  had  been  allowed  a  second 
try.  If  the  admittedly  dubious  assumptions  made  in  this 
correction  were  valid  the  results  for  the  student  and 
technician  groups  would  be  very  similar.  It  is  the  present 
writer’s  opinion  that  these  assumptions,  for  example  that 
the  probability  of  error  on  the  second  trial  would  be  the 
same  as  on  the  first,  are  so  untenable  that  the  study  is 
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entirely  inconclusive.  One  might  ask  why  a  comparison  of 
naive  high  school  students  and  seven  year  experienced 
technicians  was  made.  The  student  data  were  actually  a  part 
of  an  earlier  study  of  aptitude  and  performance.  Apparently 
it  was  decided  later  that  the  high  school  performance  should 
be  compared  to  that  of  experienced  technicians  who  were  being 
tested  using  conventional  manuals  in  order  to  generate  speed 
and  accuracy  standards.  Even  though  the  investigators' 
conclusion  that  there  was  no  substantial  difference  between 
the  groups  is  believed  unfounded,  it  is  noteworthy  that 
students  with  only  12  hours  of  training  were  able  to 
perform  as  well  as  they  did.  It  seemed  likely  that  their 
performance  would  have  been  even  better  with  a  little  more 
training.  For  instance,  the  students'  most  frequent 
error  was  component  orientation,  such  as  interchanging  the 
base  and  collector  leads  of  a  transistor.  Another  contri¬ 
bution  to  student  error  was  cold  solder  joints.  If  one 
adjusts  the  data  for  these  two  types  of  errors,  the  student 
error  rate  comes  down  almost  to  that  of  the  experienced 
technician. 


A  fully  proceduralized  fault  diagnosis  algorithm 
developed  by  the  British  Air  Force  was  tested  over  a  five 
week  period  in  1969  with  five  technicians  performing  with 
and  without  the  aid.  (Stenner  and  Robinson,  1969) .  The 
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mean  time  to  diagnose  faults  was  90  minutes  without  the 
guide  and  31  minutes  with  it.  In  nearly  half  the  non¬ 
guide  cases,  components  were  changed  in  error.  With  the 
guides  only  one  component  was  wrongly  changed  in  30  trials. 

No  statistical  tests  were  reported. 

Serendipity  Inc.,  under  contract  to  the  Naval  Air 
Systems  command,  conducted  a  field  test  of  fully  proceduralized 
non-troubleshooting  aids  involving  19  organizational  maintenance 
tasks  on  the  F-4J  in  1970  (Post,  1970) .  Fifty-two  technicians 
were  involved  with  a  total  of  635  maintenance  actions  performed. 
The  test  was  conducted  onboard  the  USS  Saratoga  and  at  the  Naval 
Air  Station,  Oceana.  Portions  of  the  existing  Maintenance 
Information  Manuals  that  were  relevant  to  the  faults  used 
in  the  test  were  collected,  checked  for  accuracy  and  packaged 
for  each  maintenance  task  so  that  the  content  of  the  traditional 
material  would  be  as  available  as  the  job  guides.  In  the  tests, 
experienced  personnel  had  the  option  of  using  or  not  using  the 
excerpts  from  the  manual?  inexperienced  personnel  were  required 
to  use  them.  Using  existing  procedures,  experienced  personnel 
performed  statisfactorily  (in  terms  of  meeting  quality  assurance 
standards)  74%  of  the  time  and  inexperienced  63%  of  the  time 
for  an  average  of  68%.  While  using  the  job  guides  experienced 
personnel  performance  was  100%  and  inexperienced  94%  for 
an.  average  of  97%.  Thus,  inexperience  personnel  performing 
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with  job  guides  were  30%  better  than  experienced  personnel 
using  existing  procedures. 

In  terms  of  procedural  errors,  experienced  personnel 
using  existing  procedures  had  14%,  with  job  guides  4%. 

The  figures  for  inexperienced  were  14%  and  6%.  This  is  an 
overall  improvement  of  64%.  All  of  the  above  results  were 
statistically  significant. 

An  additional  comparison  was  made  of  performance  over 
repeated  trials  with  and  without  the  aids  by  inexperienced  ' 
personnel  doing  operational  checks.  The  performance  with 
the  manuals  was  fast  and  stable  from  trial  to  trial,  but 
the  error  rate  was  high  and  remained  so  over  trials.  The 
error  rate  with  job  guides  was  very  low  and  the  time  to 
perform  diminished  over  trials  and  approached  the  time 
taken  without  the  aids.  It  was  apparent  that  the  manual- 
supported  performance  was  rapid  because  of  the  omission  of 
critical  steps.  The  learning  effect  found  with  the  aids 
was  similar  to  that  found. in  the  PIMO  studies. 


JPA's  were  developed  for  the  UH-1H  helicopter  in  the 
Vietnam! zation  program  for  the  Air  Force  by  XYZYX  and  were 
evaluated  in  1971.  Fifty-four  U.S.  and  36  Vietnam  tech¬ 
nicians  were  involved.  Some  were  experienced  (at  least  one 
year  experience  on  the  UH-1H} ,  some  were  apprentices 
(trained  but  little  or  no  experience) ,  and  some  were 
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novices  (no  training  or  experience) .  The  results  for 
non-troubleshooting  tasks  were  similar  to  earlier  studies. 
Experienced  USAF  personnel  did  slightly  better  with  JPA's 
than  with  conventional  documentation  (92%  completion  vs 
88%)  and  apprentices  did  much  better  (95%  vs  68%) .  For 
novices  the  figures  were  61%  vs  33%.  The  VNAF  experienced 
technicians  performed  equally  well  with  technical  manuals 
or  JPA's  and  the  VNAF  apprentices  performed  somewhat  better 
with  JPA ’ s . 

For  troubleshooting  tasks,  fully  proced'-^alized  JPA's, 
maintenance  dependency  charts  (MDC's)  and  teciaiical  manuals 
were  compared.  Performance  of  experienced  USAF  technicians 
was  better  with  JPA's  in  contrast  to  TM's  (100%  vs  83%). 

For  apprentice  js  the  figures  were  100%  vs  61%  and  for  novices 
56%  vs  44%.  For  each  group,  MDC  performance  was  inferior  to 
fully  proceduralized  JPA's  or  TM's.  Both  groups  of  VNAF 
technicians  (there  were  no  novices  tested  in  Viet  Nam) 
performed  quite  well  (80+  correct)  on  troubleshooting  tasks 
with  all  three  forms  of  documentation.  Apprentices  performed 
error  free  with  fully-proceduralized  eids  and  at  about  85% 
with  TM's  and  MDC's,  which  was  somewhat  oetter  than  experienced 
technicians. 

A  major  finding  of  the  study  was  that  the  original 
UH-1II  JPA's  were  virtually  unusable  7  ecause  of  errors.  This 
was  attributed  to  a  lack  of  a  thorough  task  analysis,  which 
had  to  be  done  to  correct  the  aids  used  in  the  study. 
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This  study  was  never  published  by  the  Air  Force  and  the 
above  information  is  from  a  two-page  summary  obtained 
informally.  It  is  consequently  difficult  to  render  any 
judgment  about  its  validity.  The  results  with  USAF 
personnel  are  consistent  with  other  experiments  except  for 
MDC's  being  inferior  for  troubleshooting.  The  results  with  , 
Vietnamese  subjects,  however,  are  confusing  and  may  be  due 
to  the  possibility  mentioned  in  the  summary  of  unavoidable 
contamination  of  some  of  the  data  obtained  in  operational  • 
environments . 

The  SAFEGUARD  Maintenance  Data  System  (MDS)  is  a  large 
(680,000  frames)  microfilm  system  designed  to  support 
maintenance  and  operation  of  SAFEGUARD.  MDS  uses  task 
oriented  step-by-step  procedures  with  illustrations  for 
linear  procedures  such  as  start  up,  adjust,  check,  lubricate, 
disassemble  and  assemble.  Troubleshooting  is  supported 
by  logic  tree  frames  and  associated  simplified  functional 
block  diagrams.  Additionally  the  system  contains  other 
support  information  such  as  signal  flow  diagrams,  repair 
parts  data,  and  location  information.  The  reader-printer 
utilized  has  a  keyboard  to  access  any  one  of  the  frames 
on  a  16  mm  rolled  film  cassette.  Each  cassette  will  have 
approximately  1,000  frames. 
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The  U.S.  Army  SAFEGUARD  Logistics  Command  has  conducted 
four  phases  of  a  five  phase  test  program  of  MDS.  The  early 
phases  were  primarily  concerned  with  data  verification  pro¬ 
cedures  and  viewing  equipment.  The  fourth  phase,  conducted 
in  October  and  November  1972,  tested  the  adequacy  and 
effectiveness  of  the  MDS  data  package  produced  by  the  Raytheon 
Company  to  support  operation  and  maintenance  tasks  on  the  Radar 
Return  Generator  (RRG) . 

Earlier  phases  of  the  test  program  used  MTTR's  as 
test  standards  with  generally  positive  results.  Phase  IV 
compared  the  performance  of  military  technicians  with  Raytheon 
engineers.  Because  of  a  number  of  differences  in  the  conditions 
under  which  the  two  groups  performed,  it  is  the  writer's 
opinion  that  the  main  results  of  the  experiment  were 
inconclusive.  However,  comparisons  of  technician  performance 
with  available  technical  logistics  data  repair  times  were 
favorable,  indicating  that  the  system  could  be  maintained 
satisfactorily  using  MDS.  Throughout  the  test  phases 
data  accuracy  was  a  problem?  in  some  cases  tests  could  not 
be  conducted  until  the  data  were  corrected. 

In  contrast  to  other  aids  developments  the  MDS  was 
designed  to  meet  stringent  MTTR  goals  with  experienced 
technicians.  MDS  is  the  only  example  of  a  large  weapon 
system  which  uses  proceduralized  aids  extensively. 
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In  1972,  the  Naval  Weapons  Engineering  Support  Activity, 
at  the  request  of  Naval  Air  Systems  Command,  tested  the 
effectiveness  of  fully  proceduralized  non-troubleshooting 
aids  that  had  been  produced  by  the  XYZYX  Corporation  for 
the  NC-8A  Mobile  electric  power  plant  (Horne,  1972).  The 
data  was  collected  over  a  six  months  period  and  consisted 
of  297  non-scheduled  intermediate  maintenance  actions  per¬ 
formed  by  26  technicians.  In  order  to  determine  the  per¬ 
formance  of  "inexperienced"  personnel,  electronics  maintenance 
actions  were  assigned  to  mechanical  technicians  and  vice 
versa.  The  results  indicate  that  inexperienced  personnel 
using  JPA's  ma^e  no  more  errors  than  experienced  personnel 
using  conventional  documentation.  Experienced  personnel 
took  longer  using  JPA's  than  they  did  with  conventional 
manuals  but  they  performed  error  free.  This  latter  result 
was  statistically  significant.  The  comparison  of  inexperienced 
technicians  using  JPA  and  experienced  technicians  using 
conventional  manuals  is  again  a  matter  of  finding  no  signifi¬ 
cant  difference  and  assuming  that  none  exists. 

Two  other  features  of  this  study  merit  comment.  Since 
the  aids  were  non-troubleshooting  ones,  inexperienced 
personnel  had  to  be  directed  to  the  procedure  to  be  followed. 

An  experienced  technician  did  the  troubleshooting  and  then 
assigned  one  of  the  inexperienced  men  a  task.  Indications 
were  that  an  experienced  troubleshooter  could  keep  at  least 
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four  inexperienced  men  occupied  doing  maintenance  using 
JPA‘s.  The  work  load  was  such  that  this  span  of  control 
could  not  be  tested;  it  may  be  wider.. 

Before  this  study  could  be  conducted  it  was  necessary 
that  the  JPA’s  be  further  validated.  They  had  been  produced 
without  a  specification  having  been  imposed  on  the  contractor 
but  they  had  been  "desk-top"  validated  before  delivery.  A 
number  of  errors  were  found  when  the  material  was  reviewed  by 
experienced  technicians  before  the  test.  However#  even  after 
the  aids  had  been  judged  to  be  complete  it  was  clear  that 
"hands-on"  validation  was  necessary  to  make  the  aids  usable 
by  inexperienced  personnel.  This  is  a  circumstance  quite 
frequently  found  during  the  review  of  JPA  studies.  As  in  a 
computer  program,  the  smallest  error  in  a  JPA  may  cause 
trouble;  because  of  their  inherent  vagueness  conventional 
manuals  escape  this  criticism. 

OTHER  DOD  PROJECTS 

In  additon  to  the.  above,  there  have  been  a  number  of 
performance  aids  developed  but. not  formally  tested.  The 
largest  number  have  been  procured  by  the  Air  Force  for 
Vietnamization;  in  addition  to  the  UH-1H  aids  were  prepared 
for  the  CH-47,  C~7A,  three  jet  engines,  a  refueling  vehicle 
and  two  fire-trucks.  JPA's  for  organizational,  inter¬ 
mediate  and  depot  maintenance  are  being  produced  for  an 
electronic  system  called  Seek  Point. 
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The  Air  Force  Systems  Command  has  developed  a  specifi¬ 
cation  (Ml'L-J-8332)  for  use  in  the  procurement  of  bilingual 
JPA's  and  an  "Americanized"  version  of  the  specification 
that  has  not  yet  been  officially  adopted.  These  specifi¬ 
cations,  which  will  be  discussed  again  later,  require  a 
process  of  careful  analysis  which  is  designed  to  avoid  the 
errors  and  lack  of  completeness  in  JPA's. 

In  the  Spring  of  1972,  the  Air  Force  conducted  a 
project  at  Altus  AFB  in  which  16  recruits  with  only  basic  ' 
training  were  given  four  weeks  of  training  in  how  to  maintain 
the  AN/APN-147  and  AN/ASN-35  navigational  radar  system  using 
fully  proceduralized  JPA's.  After  training,  the  airmen 
participated  in  a  demonstration  where  they  were  given  14 
problems.  The  report  of  the  results  of  the  demonstration  has 
not  yet  been  published  but  one  observer  has  written  that  the 
demonstration  went  well  and  that  most  of  the  visitors 
appeared  to  be  favorably  impressed. 

The  Altus  demonstration  was  a  preliminary  part  of 
an  experiment  involving  experienced  technicians  and  novices 
and  JPA ' s ,  SIMMS  aids  and  traditional  documentation.  This 
experiment,  which  will  be  disucssed  later,  has  not  been 
funded. 

The  Army  has  developed  aids  for  the  following  systems: 
armament  pad  for  aircraft,  search  light,  storage  battery, 
tank  engine,  gasoline  engine  water  pump  and  a  tow  launcher. 
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These  particular  systems  resulted  from  the  commodity  commands 
having  been  directed  to  produce  draft  aid  material  in  the 
format  specified  in  the  AFSC  specifications.  It  is  likely 
that  these  systems  were  chosen  for  their  simplicity  since 
no  funds  were  allocated  for  the  efforts.  The  final  aids 
were  developed  inhouse  by  AMC  for  all  but  one  of  the  systems; 
LTC  developed  the  other.  According  to  the  Army,  only  the 
format  standards  were  followed?  no  attempt  was  made  to  do 
the  task  analysis  called  for  by  the  AFSC  specification.  The 
aids  were  produced  from  material  in  the  technical  manuals  and 
no  "hands  on"  validation  was  done.  Continental  Army  Command 
and  Combat  Developments  Command  have  been  asked  to  evaluate 
the  aids  and  reports  are  expected  in  the  Summer,  1973. 

The  Army  scheduled  an  effort  beginning  with  a  planning 
phase  this  fiscal  year  in  which  information  presentation 
techniques  will  be  reviewed  in  the  context  of  differing 
environments,  training  levels  and  equipment  complexity. 

Aids,  including  the  possibility  of  different  combinations 
of  techniques,  will  be  developed  and  tested.  Although  a 
plan  exists  for  a  major  series  of  demonstrations  extending 
over  several  years,  funding  is  not  yet  firmed  up. 

The  Naval  Air  Systems  Command  has  procured  fully  pro- 
ceduralised  JPA's  in  accordance  with  the  AFSC  specifi¬ 
cation.  for  four  subsystems  of  the  LAMPS,  a  destroyer  based 
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helicopter.  These  subsystems  were  chosen  because  they  are 
high  maintenance  areas  and  present  training  problems.  A 
total  of  $100,000  was  spent  with  three  contractors,  Kentron, 
a  subsidiary  of  LTV,  XYZYX  and  Westinghouse.  All  but  one 
of  the  systems  have  been  completed  and  are  being  verified 
on  a  sampling  basis  only,  because  of  lack  of  funds. 

The  Naval  Air  Systems  Command  has  in  process  the 
development  and  test  of  fully  proceduralized  JPA's  for  the 
AQA-7  sonar  subsystem.  A  12-week  training  program  based  on 
the  JPA's  will  be  developed  and  two  inexperienced  groups 
differing  in  aptitude  will  be  trained  to  maintain  the 
equipment.  A  field  demonstration  will  then  be  made. 

Navair  has  also  sponsored  research  on  innovative  job 
aid  concepts  (Post  and  Price,  1972).  The  investigators 
point  out  that  most  research  in  job  aids  has  concentrated 
on  ways  to  assist  users  at  a  given  level  of  aptitude  and 
experience  to  perform  in  minimum  time  consistent  with 

accuracy  requirements.  They  argue  that  job  aids  "can  be 

>  * 

made  to  impart  knowledge  and  to  permit  the  accumulation  of 
experience  with  the  intention  of  facilitating  achievement 
of  specific  career  plateaus."  These  investigators  suggest 
a  number  of  ways  in  which  JPA's  might  be  improved  to  be  more 
useful  at  different  experience  and  aptitude  levels. 

Progress  here  would  alleviate  one  of  the  problems  with 
JPA's.  Experienced  personnel  do  not  heed  or  like  the  detail 
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that  is  required  by  the  inexperienced.  The  level  of  training 
and  experience  of  the  personnel  that  will  be  using  the  aids 
is  an  important  and  recognized  input  to  the  process  of  JPA 
development,  but  there  has  been  little  attention  to  a  single 
aid  or  set  of  aids  for  use  by  technicians  of  different  skill 
levels . 

In  connection  with  this  research  Navair  plans  to  conduct 
a  field  test  in  1973  comparing  an  MDC  that  was  prepared  for 
the  AWG-10  avionics  system  by  Westinghouse  with  an  innovative 
aiding  concept  that  has  emerged  from  the  research.  The  study 
will  use  both  specialists  and  apprentices  and  will  attempt 
to  relate  experience  level  to  the  effectiveness  of  each  aid. 
NON  DOD  DEVELOPMENTS 

An  attempt  was  made  to  assess  non-DOD  efforts  in  the 
job  aid  field.  There  are  a  number  of  instances  where  highly 
proceduralized  aids  are  used  in  production  but  their  use  in 
maintenance  does  not  seem  to  be  widespread.  Of  a  number 
of  electronics  companies  contacted  only  Motorola  appeared  to 
have  adopted  job  aid  concepts  in  maintenance.  They  use 
symptom  charts,  functional  block  diagrams  and  logic-trees 
in  their  television  manuals  and  in  some  cases  use  JPA-like 
step-by-step  procedures. 

Of  great  interest  is  the  use  of  the  general  job  aid 
concept  among  the  airlines.  Douglas,  Lockheed  and  Boeing 
have  each,  for  their  jumbo  jets,  developed  job  aids 
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for  flight  line  maintenance .  The  Douglas  system  for  the 
DC-10  is  called  FEFI/TAFI  (Flight  Environment  Fault 
Indication/Turn  Around  Fault  Isolation) ;  the  Lockheed  L-1011 
system  is  FIRM  (Fault  Isolation  Reporting  Method) ;  and  the 
Boeing  747  system  is  MRS/S RM  (Malfunction  Reporting  System/ 
Specific  Repair  Methods) .  Each  method  involves  the  assign¬ 
ment  of  a  code  to  a  particular  pattern  of  symptoms  as  it 
appears  in  the  aircraft.  This  code  can  be  radioed  ahead 
and  related  to  the  ground  portion  of  the  documentation,  which 
contains  procedures  for  isolating  the  fault.  The  FEFI/TAFI 
material,  the  only  material  examined  by  the  writer,  appears 
to  be  similar  to  the  JPA  as  developed  by  AFHRL  but  prepared 
for  a  higher  experience  level.  Just  how  extensive  an 
analysis  was  performed  is  not  clear.  Douglas  reports  that 
all  the  troubleshooting  information  is  being  included  in 
TAFI  but  they  are  limiting  the  FIFI  portion  to  critical 
functions  that  involve  cockpit  indicators  so  that  the  flight 
engineer  will  not  be  burdened  with  excessive  documentation. 

A  number  of  airlines  were  contacted  concerning  their 
impression  of  the  three  systems.  Most  of  the  airlines 
used  the  system  as  furnished  by  the  manufacturer.  At  least 
in  the  Douglas  case,  there  is  little  choice  since  FEFI/TAFI 
is  the  troubleshooting  portion  of  the  documentation.  The 
most  frequent  criticism  involved  the  symptoms  coding. 

Many  common  problems  don’t  have  codes.  The  airlines 
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recognize  that  the  system  can  be  and  is  being  improved  as 
flight  experience  is  gained,  however.  Another  concern  is 
that  flight  engineers  don't  always  have  time,  or  take  time, 
to  do  the  coding.  There  was  no  indication  that  the 
technique  has  led  to  the  use  of  less  skilled  personnel, 
although  one  statement  was  made  that  mechanics  not  familiar 
with  the  airplane  found  it  helpful,  particularly  the  unit 
location  information. 

In  their  literature  Douglas  (Adams  and  Boyer,  1970) 
asserts  that  FEFI/TAFI  represents  a  90%  improvement  in 
accuracy  and  that  fault  isolation  time  is  decreased  by 
more  then  66%. 


39 


DISCUSSION  OF  PERFORMANCE  AIDS 

# 

DIFFICULTY  OF  COMPARATIVE  STUDIES 

Most  of  the  evidence  on  the  effectiveness  of  job  per¬ 
formance  aids  has  derived  from  comparisons  with  "traditional" 
documentation.  It  has  been  well  established  by  surveys 
(Folley  and  Elliot/  Serendipity,  1969)  and  observations 

in  the  field  that  technicians  often  do  not  use  technical 
manuals,  even  when  available,  in  carrying  out  maintenance 
functions.  Thus,  the  studies  cited  above  (with  two 
exceptions  discussed  below)  were  comparisons  of  JPA's  vs 
the  normal  behavior  of  experienced  personnel  which  often 
meant  performance  without  documentation.^/  A  serious 
implication  of  this  is  that  it  is  not  known  whether  the 
experimental  aid  being  tested  was  the  cause  of  the  demon¬ 
strated  improvement  or  whether  improvement  occurred  because 
technicians  performing  with  the  aid  had  relevant  and 
accurate  information  available  in  some  form. 

The  Naval  Air  Systems  Command's  Maintenance  Information 
Automatic  Retrieval  System  (MIARS)  incorporates  a  number  of 
features  that  will  increase  the  utility  of  traditional  docu¬ 
mentation.  It  is  a  microfilm  system  that  will  utilize  reader- 
printers  and  have  the  capability  of  producing  hardcopy  manuals. 
The  content  of  the  system  is  being  developed  under  a  "work 
package"  concept  that  includes  some  of  the  features  found  in 

1/The  studies  also,  of  course,  involved  the  use  of 

inexperienced  technicians:  this  will  be  discussed  later. 
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the  new  aids.  Planned  for  inclusion  are  simplified  Maintenance 
Dependency  Charts,  the  principal  feature  of  SIMMS  and  the 
troubleshooting  portion  of  PIMO,  and  logic  flow  charts.  Work 
packages  are  to  be  approximately  30  pages  in  length  and  are 
to  contain  all  needed  information,  including  Illustrated 
Parts  Breakdowns,  parts  lists  and  support  equipment  needed 
for  a  particula  task  instead  of  having  these  data  scattered 
among  several  documents.  An  advanced  system  for  updating  is  also 
planned  and  in  the  process  of  being  procured  (TRUMP  -  Technical 
Revision  and  Update  of  Manuals  and  Publications) . 

Probably  everyone  associated  with  the  maintenance 
documentation  problem  would  agree  that  the  above  approach 
would  be  an  improvement.  But  how  close  to  "optimum"  is  such 
a  system?  One  feature  that  would  probably  improve  MIARS  is 
random  access  coding  of  the  microfilm  version  of  the  docu¬ 
mentation.  The  Navy  asserts  that  the  coding  and  the  requisite 
sophisticated  readers  are  too  expensive  and  that,  with  the 
work  package  organization  of  data,  random  access  will  not 
be •required.  The  McDonnell-Douglas  "WSMAC"  (Weapons  System 
Maintenance  Action  Center)  now  in  operation  at  the  St.  Louis 
plant  is  a  binary-coded  access  system  with  logic  tree 
troubleshooting  and  fault  isolation  aids.  It  was  reviewed 
by  the  Navy  MIARS  people  and  rejected  as  too  expensive. 

The  multipurpose  technical  documentation  system  that  has 
come  about  (one  hesitates  to  say  evolved)  since  before  World 
War  II  is  inadequate  for  many  reasons  -  inaccurate,  usually 


41 


0 

1j 

L! 

1; 

> 

i_ 

t 

L 

! 


Li 

Li 


i 


f  i 


obsolescent,  often  incomplete,  so  bulky  and  organized  in 
such  a  fashion  that  it  is  used  in  the  field  only  as  a  last 
resort.  It  can  hardly  be  characterized  as  a  system  that  only 
needs  to  be  fine-tuned.  Consequently,  it  is  difficult  to 
assess  the  improvement  that  will  result  from  the  changes,  such 
as  MIARS,  that  are  now  being  introduced  by  the  Service. 

Do  these  rather  conservative  changes  (which  are,  in  one 
form  or  another,  also  present  in  the  "new"  job  aids)  account 
for  the  improved  performance  of  experienced  technicians  when 
they  use  the  new  aids?  To  cite  a  specific,  one  research 
investigator  conjectures  that  the  efficiency  of  the  PIMO 
approach  probably  is  due  to  the  use  of  readibility  standards 
and  human  factors  task  analysis,  features  that  should  be 
and  sometimes  are  included  in  conventional  documentation 
(Booher,  1972) . 

Some  of  the  comparative  studies  have  included  an  effort 
to  insure  that  the  conventional  documentation  was  accurate, 
but  to  the  writer's  knowledge  there  have  been  only  two 
studies  where  the  reference  system  of  documentation  while 
still  "conventional"  has  been  brought  up  to  the  standard  of 
the  experimental  system  in  terms  of  accuracy,  completeness, 
compactness  and  readability,  characteristics  obtainable 
within  the  evolving  conventional  system.  One  of  the 
exceptions  was  the  AFHRL  study  which  compared  fully  proce- 
duralized  JPA's  and  conventional  documentation  on  the 
Maintenance  Task  Simulator  (MTS-2)  (Elliott,  AFHRL-TR-68-1) . 
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Here,  TO-like  documentation  was  produced,  since  manuals  did 
not  exist,  which  included  schematics,  theory  of  operation, 
system  geography  information  and  voltages,  resistances 
and  waveshapes  at  selected  points.  In  the  other  study, 
Serendipity's  F4-J  experiment,  a  deliberate  effort  v;as  made 
to  optimize  the  conventional  documentation.  Relevant 
portions  of  the  manual  were  collected  for  each  maintenance 
action,  checked  for  accuracy  and  convencntly  packaged. 

As  indicated  in  the  discussion  of  the  experiments,  the 
JPA's  we re  still  superior. 

The  above  discussion  should  not  be  construed  as  a 
recommendation  that  DOD  be  satisfied  with  the  small  steps 
that  are  being  taken  to  improve  maintenance  documentation? 
there  is  a  clear  need  for  better  manuals  in  general  but  it  is 
not  yet  clear  which  of  several  alternatives  will  best  serve 
the  experienced  technician. 

While  the-  evidence  with  respect  to  job  aids  for  the 
experienced  technician  is  eqn".. vocal,  such  is  not  the  case 
with  less  experienced  or  apprentice  personnel.  In  all  of  the 
experiments  and  field  tests,  inexperienced  technicians 
performed  better  with  fully  proceeuralized  aids  than  with 
conventional  documentation.  Often,  particularly  in  trouble¬ 
shooting  situations,  the  inexperienced  technicians,  even 
those  who  had  attended  the  prescribed  schools,  were  unable 
to  perform  at  all  using  conventional  manuals.  However, 
they  were  able  to  perform  with  minimal  errors  using  the  aids 
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and  their  time  to  repair  approached  the  time  experienced 
technicians  required  when  the  latter  used  either  conventional 
or  experimental  documentation. 

There  is  no  reason  to  believe  that  documentation 
improvements  now  being  made  will  effect  any  significant  change 
in  the  performance  of  the  inexperienced  technician.  It  is 
true  that  step  by  step  procedures  are  included  in  manuals 
for  some  maintenance  tasks  and  would  presumably  be  at  hand 
in  the  improved  conventional  manual;  however,  they  are  not 
ordinarily  based  on  a  task  analysis  of  the  type  needed  and' 
are  not  explicit  and  detailed  enough  to  be  used  by  the 
novice  without  guidance  from  an  experienced  technician. 
Considering  the  overwhelming  evidence  from  the  experiments 
and  field  tests  cited  above,  it  is  the  writer's  conviction 
that  non-troubleshooting  fully  proceauralized  job  perfor¬ 
mance  aids  should  be  produced  and  widely  used.  Such  an 
action  would  result  in  significant  savings  in  the  cost  of 
maintenance  in  that  the  Services  would  be  able  to  use 
more  effectively  the  new  technician  who,  although  he  has 
presumably  been  properly  trained,  spends  many  months  on 
the  job  before  he  performs  any  but  the  simplest  tasks. 

Even  granting  that  JPA's  may  be  somewhat  more  expensive 
to  produce,  (an  issue  treated  later)  the  savings  due  to  more 
effective  use  of  new  technicians  would  be  significant  over 
the  life  of  a  new  weapon  system. 
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INSTITUTIONAL  CHANGE  PROBLEM 

Achieving  efficient  cost  effective  maintenance  is  a 
system  problem  involving  management  procedures,  procurement 
methods,  maintenance  philosophy,  and  training  and  personnel 
practices.  The  system  includes  a  number  of  well  established 
institutions  with  all  of  the  problems  inherent  in  a  complex 
process  that  has  developed  over  a  long  period  of  time. 

Perhaps  the  major  institutional  problem  is  the  relation 
of  life  cycle  cost  and  the  system  acquisition  process. 
Inadequate  weight  is  given  to  the  potential  cost  savings 
in  maintenance  during  the  initial  decision  on  what  kinds 
of  documentation  to  procure  for  the  new  system.  Another 
part  of  the  problem  is  in  the  contractor  side  of  the  docu¬ 
mentation  community.  Traditional  documentation  is  understood; 
it  is  easier  to  estimate  and  produce.  Given  the  low 
priority  and  status  documentation  activities  have  inside 
the  major  weapons  system  firms,  it  isn't  surprising  that 
there  is  little  pressure  for  change.  It  seems  evident  that 
the  impetus  for  change .will  only  come  from  those  having  broad 
enough  responsibility  to  be  concerned  with  the  whole  life 
cycle  process. 

PERSONNEL  AND  TRAINING  IMPLICATIONS 

In  addition  to  the  benefits  derived  from  less  costly 
maintenance  and  improved  equipment  reliability,,  job  aids 
offer  the  possibility  of  greatly  reducing  training  costs. 

In  general  each  of  the  Services  follow  the  same  approach 
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leading  to  the  assignment  of  a  man  to  an  organization  level 
maintenance  position  where  he  is  concerned  with  a  given 
system,  for  example,  an  electronics  system.  After  basic 
training,  the  individual  is  sent  to  a  school  and  given  an 
extensive  course  in  basic  electronics  with  emphasis  on 
general  theory.  He  subsequently  receives  shorter  training 
in  the  theory  and  maintenance  of  a  particular  system  or 
systems  for  a  total  of  30  or  40  weeks  of  formal  training.  He 
then  is  assigned  to  a  maintenance  organization,  and  is  supposed 
to  be  capable  of  productive  work  with,  of  course,  the  assumption 
that  he  will  be  given  additional  training  through  OJT. 

Many  experienced  observers  indicate  that  in  practice  the 
system  doesn't  work  this  way.  The  new  technician  is  unable 
to  use  the  conventional  manuals  and  must  be  closely  super¬ 
vised  if  he  does  any  but  the  most  mundane  maintenance  task. 

Since  on  the  job  training  is  often  inadequate  he  learns 
only  if  he  is  highly  motivated  and  working  with  experienced 
personnel  willing  and  able  to  teach  him.  Given  the 
reenlistment  rate  (currently  11%  for  AF  Electronic  Technicians) 
and  the  length  of  training,  the  cost  of  effective  maintenance 
finally  obtained  from  this  process  is  extremely  high.  The 
JPA  and  training  research  literature  is  replete  with  . 
studies  indicating  that  even  lower  aptitude  mer*  can  carry 
out  non-troubleshooting  maintenance  tasks  with  minimal 
training  if  they  are  furnished  with  properly  prepared 
job  performance  guides.  If  no  change  were  made  in  formal 
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training  and  personnel  procedures,  significant  improvement 
would  still  result  from  providing  the  newly  assigned 
technician  JPA's,  if  only  on  selected  subsystems,  since  he 
would  immediately  be  able  to  be  productive. 

Even  greater  economy  would  result  if  the  training 
procedures  were  changed.  With  four  to  six  weeks  of  training 
in  the  use  of  tools  and  support  equipment  the  average 
recruit  should  be  able  to  perform  useful  work  using  JPA's 
(Foley,  1972) .  There  would  be  substantial  and  valid 
objections  if  this  were  the  only  change  made.  Even  though' 
the  evidence  indicates  new  technicians  derive  a  good  deal 
of  job  satisfaction  from  actually  doing  productive  work 
with  JPA's,  effective  OJT  must  be  provided  if  they  are  to 
remain  motivated  and  to  increase  their  effectiveness.  With 
the  exception  of  some  of  the  earlier  studies  (Forecast, 
JOBTRAIN,  etc.)  job  aids  design  has  not  included  training 
objectives.  A  number  of  studies  (PIMO,  F4-J  Study,  HRL  Study) 
indicate  that  training  nevertheless  does  occur.  Research 
by  Post  and  Price  indicates  that  modified  JPA's  can  be  an 
effective  OJT  vehicle,  one  that  would  be  relatively  inde¬ 
pendent  of  the  teaching  abilities  and  inclinations  of  the 
personnel  ordinarily  charged  with  OJT  responsibility 
(Post  and  Price,  1972).  While  training-oriented  JPA's 
would  probably  be  more  effective  than  the  usual  OJT, 
additional  provisions  would  be  needed  for  career  progression 
for  those  technicians  exhibiting  the  talent  and  motivation 
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for  advancement.  Such  personnel,  should  they  choose  -CO 
reenlist,  could  be  sent  to  school  after  a  tour  of  hands-on 
maintenance  experience,  a  system  used  by  the  British 
Navy  (Huggett,  1970) .  Technicians  who  do  not  stay  in  the 
service  would  likely  enter  civilian  life  better  trained 
than  under  the  present  system  since  they  now  retain  little 
or  nothing  of  the  original  formal  training  received  before 
beginning  their  military  maintenance  careers  and  spend  a 
relatively  short  time  actually  at  work. 

Another  problem  with  the  current  training  and  assignment 
philosophy  is  the  difficulty  and  expense  of  cross  training. 
Foley  (Foley,  1972)  points  out  for  example  that  there  are 
34  systems  that  are  the  responsibility  of  Air  Force  Specialty 
codes  328X4  (Avionic  Inertial  and  Radar  Navigation  Systems 
Specialists) .  The  formal  training  for  this  career  field 
is  37  weeks  long  with  24  additional  weeks  of  OJT.  After 
this  training  the  technician  is  effective  only  on  the 
system  for  which' he  has  received  OJT.  When  reassigned, 
extensive  and  expensive  cross  training  is  required  before  the 
technician  is  again  effective.  Job  performance  aids  offer 
the  potential  for  significantly  reducing  the  cost  of  cross 
training. 

COST  SAVINGS  .POTENTIAL 

Hard  evidence  of  the  cost  savings  possible  from  adopting 
JPA's  is  difficult  to  develop.  In  the  F4-J  study  (Post, 

1970)  a  simple  mathematical  model  of  a  work  center  was  built 
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to  explore  the  manpower  utilization  implications  of  introducing 
job  guides  for  non-troubleshooting  tasks.  Without  job 
guides  71%  of  the  inexperienced  labor  is  spent  observing 
and  assisting  experienced  technicians.  The  other  29% 
cannot  bo  accommodated  by  the  work  load.  If  job  guides 
were  used  83%  of  the  inexperienced  would  be-  performing 
maintenance  and  the  other  17%  assisting.  The  availability 
of  experienced  technicians  for  complex  work  would  increase 
52%;  the  maintenance  queue  would  decrease  by  25%  and -the  • 
number  of  maintenance  actions  failing  quality  assurance 
would  decrease  by  75%. 

In  the  PIMO  study  system  effectiveness  estimates  were 
also  made.  It  is  pointed  out  that  simply  utilizing  time 
now  spent  in  OJT  for  maintenance  using  job  guides  would 
decrease  the  time  a  one  term  enlistee  spends  in  training  by 
25%.  The  improved  performance  expected  with  job  guides 
could  improve  departure  reliability  by  50-65%  and  operational 
readiness  time  by  38-40%.  If  such  performance  measures 
were  kept  at  pre-PXMO  levels,  a  reduction  in  unscheduled 
maintenance  manpower  of  30-39%  would  be'  possible. 

Sould  JPA’s  be  widely  adopted  and  formal  technical 
training  reduced,  very  significant  savings  could  be  made  in 
training  costs.  The  PIMO  report  statos  that  there  were  in 
1969  approximately  5, 700  aircraft  mechanics  assigned  to 
flight  line  maintenance  of  the  C-141A.  Assuming  only  1,000 
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new  men  per  year  due  to  turnover/  and  a  per  man  cost  of 
$3/500  for  OJT  and  $4/500  for  formal  training/  over 
$7,000,000  per  year  could  be  saved  in  this  specialty  by 
cutting  training  from  28  to  4  weeks. 

About  one-fourth  of  total  DOD  enlisted  strength  is  in 
maintenance.  Of  this  force  of  about  600,000  more  than  one- 
fourth,  or  150,000,  must  be  replaced  each  year  because  of 
turnover.  If  the  PIMO  calculations  are  extended  to  this 
entire  maintenance  fcrce  the  savings  would  be  nearly  $1.2 
billion  per  year.  Such  an  extension  is  probably  not  war¬ 
ranted  since  JPA's  are  not  appropriate  for  every  maintenance 
task.  On  the  other  hand,  this  is  only  the  potential  cost 
benefit  from  reducing  training  and  does  not  include  the  benefit 
from  better  equipment  availability  and  fewer  zero  defect  removals. 

If  these  studies  are  at  all  reasonable  the  cost  benefit 
potential  of  JPA's  is  indeed  very  large,  but  it  is  obtainable 
only  by  making  changes  that  cut  across  the  decision  making 
structure  of  the  services  and  by  making  investments  that 
do  not  clearly  accrue  to  the  benefit  of  the  investing  agency. 

One  objection  to  JPA's  has  been  that  they  cost  more 
than  conventional  documentation  and  that  project  managers 
faced  with  competing  requirements  resist  their  adoption. 

If  the  cost  savings  potential  of  JPA's  is  even  a  small 
fraction  of  what  is  claimed  by  proponents,  the  initial  cost 
should  not  be  an  overriding  factor.  A  budget  quotation 
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submitted  to  AFLC  in  1971  for  completion  of  flight  line  JPA's 

for  the  C-141  was  $1,300,000  with  troubleshooting  aids 

and  $800,000  without  (Ostrominski ,  197.2).  McDonnell-Douglas 

reportedly  estimated  that  JPA's  for  the  P-15  would  cost 

$45  million  versus  $35  million  for  conventional  documentation; 

Most  estimates  indicate  the  cost  at.  100%  to  125%  of 
conventional  documentation.  In  at  least  one  case,  Seek  Point, 
JPA  cost  estimates  were  less  than  the  estimates  for  con¬ 
ventional  manuals  * 

If  JPA’s  were  widely  adopted  the  production  costs  would 
undoubtedly  come  down.  Current  JPA  estimates  from  contractors 
accustomed  to  producing  conventional  manuals  are  probably 
inflated  because  of  uncertainty.  The  industrial  base  for  this 
.kind  of  product  would  expand  although  fortunately  there  are 
currently  at  least  a  half  dozen  contractors  who  have  demon¬ 
strated  capability  in  this  area. 

TASK  ANALYSIS 

Another  problem  associated  with  acceptance  of  JPA's  is 
that  it  is  not  generally  recognized  that  performance  aid 
development  is  an  evolving  technology  and  that  today's  aids 
are  improvements  over  earlier  ones.  PIMO,  perhaps  the 
widest  known  and  around  which  there  has  been  controversy, 
started  in  1964.  Since  then  the  process  of  performing 
maintenance  task  .analyses  has  been  formalized  with  a 
consequent  improvement  in  the  resulting  product;  Job  or 
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task  analysis  is  an  old  concept  which  has  been  used  for  a 
variety  of  purposes.  What  is  labeled  task  analysis  can  vary 
from  the  superficial  survey  to  the  most  thorough  and  detailed 
analysis  of  the  factors  that  influence  how  a  particular  task 
is  done.  The  JPA  task  analysis  is  of  the  latter  type.  Some 
of  the  factors  that  are  included  are  the  type  of  equipment/ 
the  level  of  maintenance  to  which  tasks  are  assigned/  the 
type  of  test  equipment,  the  aptitude  of  the  user  and  what  he 
is  expected  to  have  learned  during  training.  These  factors 
are  included  in  present  procedures  such  as  the  Army’s 
Maintenance  Engineering  Analysis.  The  emphasis  in  the 
task  analysis,  however,  is  not  on  what  is  to  be  done  to  the 
equipment  but  on  how  the  technician  is  to  do  it. 

In  order  to  insure  that  the  delivered  product  is  complete 
and  accurate,  Air  Force  Systems  Command  has  developed  a 
specification,  Mil-J-8332  mentioned  earlier,  which  requires 
certain  subproducts,  such  as  a  task  inventory,  that  are 
meant  to  insure  the  quality  of  the  final  product  and  which 
can  only  be  produced  if  the  detailed  task  analysis  is 
performed.  Some  of  the  JPA's  that  are  well  known  and  that 
have  influenced  the  acceptability  of  the  approach,  such  as 
those  for  the  UH-1H,  were  produced  before  this  specification 
was  developed.  It  is  expected  that  the  specification  and 
accompanying  handbooks  will  improve  the  quality  of  JPA’s. 
Another  specification,  Mil-u-38800,  which  has  been  developed  by 
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AFLC,  does  not  require  the  subproducts  and  thus  does  not 
require  detailed  task  analysis.  The  writer  is  in  agreement 
with  the  proponents  of  JPA's  that  products  procured  using 
this  specification,  while  having  the  same  format  and 
appearance,  will  not  be  as  effective  as  those  based  on  task 
analysis  since  even  small  errors  can  render  the  aids  virtually 
unusable . 

NEED  FOR  DEFINITIVE  DEMONSTRATION 

Considering  the  potential  of  maintenance  aids  for  less 
expensive  maintenance,  better  equipment  availability  and 
better  manpower  utilization  it  is  difficult  to  understand 
why  more  money  has  not  been  spent  in  bringing  this  approach 
to  a  point  where  a  clear-cut  decision  as  to  its  merit  could 
be  made.  The  PIMO  project  cost  $2.8M.  All  of  the  AFHRL 
effort  on  JPA's  since  1960  cost  a  total  of  $540K.  The  early 
Arm\  HumRRO  work  on  performance  aids  was  estimated  to  be  less 
than  $1. 0M.  The  Navy  expenditures  on  research  in  the  area 
are  about  $500K  to  date.  Thus  approximately  $5M  has  been 
spent  developing  and  evaluating  something  that  shows  promise 
of  saving  many  times  that  amount  annually. 

The  numerous  small  research  studies  over  the  years  have 
not  had  the  influence  that  maintenance  aid  proponents  have 
hoped  for.  Because  of  the  general  difficulties  surrounding 
human  factor  field  research  and  the  small  sample  sizes 
found  in  the  aid  studies,  a  number  of  doubts  remain, 
particularly  in  the  troubleshooting  area.  There  are 
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differences  of  opinion  ever,  among  job  aid  proponents  about 
the  optimum  way  to  present  maintenance  data  and  there  are  a 
number  of  issues  needing  more  study,  some  of  which  have  been 
discussed  in  this  report. 

In  the  writer's  opinion  it  is  time  to  "freeze  the  design" 
and  demonstrate  the  product..  A  large  scale  series  of  well- 
planned  field  demonstrations  of  job  aids  should  be  made 
with  the  active  involvement  of  all  parties  concerned  with 
maintenance  documentation.  These  should  be  funded  well 
enough  to  insure  that  the  results  will  be  compelling  and  if 
positive  will  lead  to  acceptance  and  implementation. 

The  Air  Force  Human  Resources  Laboratory  has  proposed 
in  Project  Innovate  (Project  1194  of  Program  Element  63102F) 
an  effort  which,  with  some  modification,  would  serve  part  of 
this  purpose.  This  project,  which  has  not  been  funded, 

* 

«► 

would  compare  conventional  technical  orders,  SIMMS  or  MDC 
type  decision  aids  and  fully-proceduralized  JPA's  and 
would  provide  answers  to  the  questions  of  whether  brief 
technical  training  is  adequate  with  JPA's  and  whether  JPA's 
alleviate  the  cross-training  problem  within  a  specialty. 

It  would  also  compare  performance  of  personnel  of  differing 
aptitude  levels. 

The  project  would  consist  of  a  number  of  phases: 
development  of  JPA's  for  all  or  at  least  a  representative 
subset  of  equipments  maintained  by  personnel  of  one  electronic 
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Air  Force  Specialty  Code  (AFSC) ,  the  development  of  short 
simplified  training  programs  to  accompany  the  aids,  the 
development  to  job-task  performance  tests  to  be  used  in 
evaluation,  and  the  conduct  of  controlled  experiments  and 
demonstrations.  As  indicated  earlier  a  preliminary  try  out 
of  aids  and  a  training  program  for  the  AN/APN-147  doppler  radar 
and  its  computer,  the  AN/ASN-35  was  held  in  1972  at  Altus 
Air  Force  Base.  Using  these  equipments  the  first  controlled 
experiment  will  compare,  at  the  orgnization  and  intermediate 
levels  of  maintenance,  fully  proceduralized  aids,  SIMMS  decision 
aids  and  technical  orders.  A  later  experiment  will  determine 
whether  men  briefly  trained  to  use  JPA's  on  one  equipment  can 
effectively  maintain  unfamiliar  equipment  in  the  same  AFSC 
using  JPA's.  A  positive  finding  here  would  have  great  signifi¬ 
cance  since  technicians  must  currently  be  given  extensive 
OJT  on  the  new  equipment  before  they  are  effective,  even  though 
they  have  been  given  formal  theoretical  training  to  make  the 
transition  easy.  These  demonstrations  will  be  evaluated  on  a 
number  of  criteria:  performance  on  job  task  performance  tests 
(not  paper  and  pencil  theory  tests) ,  spare  parts  consumption, 
manpower  requirements  and  training  costs.  The  Air  Force  plans 
to  use  different  contractors  for  conducting  and  evaluating 
the  demonstrations.  The  total  program  will  extend  over  five 
years  and  will  require  approximately  $2,0M  as  currently 
planned. 
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INFORMATION  PRESENTATION  MEDIA 


Up  to  this  point  this  report  has  been  primarily  concerned 
with  the  content  and,  to  a  lesser  extent,  format  of  maintenance 
documentation.  The  methods  of  presentation  to  the  user, 
including  methods  of  keeping  the  material  current,  are  also 
important.  This  is  particularly  true  for  major  problems  in 
maintaining  up  to  date  information  and  accessing  the  material 
needed  to  troubleshoot  and  repair  complex  subsystems. 

There  have  been  numerous  DOD  and  industry  studies  of 
documentation  and  while  most  of  them  have  been  concerned  with 
management  problems  some  have  concerned  themselves  with 
method  of  presentation,  one  of  the  most  recent  being  a  NSIA 
study  for  the  Air  Force  (NSIA  1572)  which  recommended  the 
use  of  microfilm  for  documentation,  including  flight  line 
maintenance . 

Although  no  attempt  will  oe  made  here  to  review  these 
studies,  a  few  general  comments  can  be  made.  Each  of  uhe 
services  is  moving  to  microfile  systems  of  one  sort  or  the 
other.  Considering  industry  experience,  for  most  applications 
the  conversion  can  be  justified  on  economic  ground  alone 
even  if  nothing  more  is  done  than  copy  the  existing  manuals. 
Much  of  the  potential  of  microform  will  of  course  not  be 
realised  if  this  is  all  that  is  done.  The  MIARS  system 
which  has  been  referred  to  earlier  is  an  attempt  to  gain 
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by  providing  for  rapid  up-dazing.  The  SAFEGUARD  MDS  not  only 
does  this  but  also  incorporates  fully  proceduralized  aids. 

Such  systems  are  well  within  the  state-of-the-art. 

The  USMAC  system  developed  by  McDonnell  Douglas  as  a 
candidate  for  F-15  technical  documentation  is  a  binary  coded 
microfilm,  system  which  includes  trouble  shooting  trees  ana 
symptom  indexes  as  well  as  conventional  documentation. 

Although  this  system  was  not  accepted  for  the  F-15/  it  is  in 
use  at  the  McDonnell  Douglas  plant  at  St.  Louis  for 
acceptance  testing  and  trouble  shooting  on  new  aircraft.  It 
currently  contains  a  large  fraction  of  F-4  technical  manual 
materials.  The  binary  code  access  permits  heirarchical 
retrievals  on  materials  related  to  a  subsystem  or  component 
to  the  depth  desired. 

As  mentioned  earlier  random  access  coding  in  MIARS  was 
judged  to  be  unnecessary  and  too  expensive.  Should  more 
sophisticated  systems  be  developed?  Is  there  a  need  for 
centralised  data  banks  with  remote  terminals?  Considering 
the  potential  for  improvement  in  MIARS  or  MDS-  level  systems 
and  in  the  updating  speed  and  accuracy  that  computer  output 
microfilmers  will  make  possible,  it  is  doubtful  that  centralised 
data  banks  and  remote  terminals  will  be  needed  to  provide 
maintenance  data  that  is  sufficiently  current.  There  may  be 
economic  benefits,  however,  that  should  be  explored. 

There  are  other  reasons  however  to  consider  central  data 
banks  with  remote  terminals.  The  possibility  of  the  maintenance 
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man  being  able  to  interact  with  a  computer  on  a  real  time  !• 

basis  as  he  troubleshoots  deserves  investigation .  Such  a  i 

capability  could  also  be  used  to  provide  computer  assisted  i 

) 

instruction  (CAI)  to  maintenance  personnel. 

^Audio-visual  systems,  often  earphones  and  small  slide  i 

projectors,  have  been  used  successfully  for  years  in  manu¬ 
facturing  production  lines  for  presenting  assembly 
instructions. 

The  few  DOD  studies  in  audio  visual  approaches  to  pre¬ 
senting  maintenance,  data  have  not  been  positive,  in  part 
because  of  the  nature  of  the  equipment  used.  The  literature 
suggests,  however,  that  audio  visual  presentation  of  information 
can  be  effective  providing  care  . s  taken  in  what  is  presented 
in  each  mode  (Post  and  Price,  1572).  Recent  applications 
of  video-  tape  systems  to  training  have  yielded  promising 
results.  For  example,  a  series  of  ten  color  video  tapes 
each  about  20  minutes  long  has  been  produced  by  Learning  1 

Achievement  Inc.  for  Burroughs  to  be  used  in  teaching  computer 
operation  and  operator  maintenance.  The  system  is  interactive: 
in  that  the  student  stops  or  rewinds  the  machine  and  answers 
questions  in  a  workbook.  The  use  of  such  an  approach  to  convey 
an  overall  picture  of  an  operation  -  something  a  JPA  does 
not  do  -  or  to  introduce  retrofits  might  be  effective. 


INFORMATION  CONCEPTS  INCORPORATED 


■ETC 

ILsj 


r  -  •  *  -  •  Tr~-  i  rai  ifMirhii^— - d ^  i • 


RECOMMENDATIONS 


System  Demonstration 

There  is  a  need  for  a  program  of  definitive  demonstrations 
to  settle  the  main  issues  in  the  area  and  to  provide  the 
needed  impetus  for  .'implementation  of  those  features  that  are 
effective  in  a  ~e«l  operating  environment.  The  needed  demon¬ 
strations  should,  be  conducted  with  each  of  the  military 
departments  being,  involved. 

(1)  The  ^proposed  but  unfunded  Air  Force  project 
"Innovate"  should  be  enlarged  ana  carried  out.  This  project 
will  compare  conventional  technical  orders,  decision  aids 
and  fully  procedural i zed  job  performance  aids  (JPA's)  and 
will  provide  answers  to  the  questions  of  whether  brief 
technical  training  is  adequate  with  JPA's  and  whether  JPA's- 
alleviate  the  cross-training  problems  within  a  specialty. 

(2)  The  Army  Vs  "Low  Cost  Ownership"  Program  on  the 
improvement  of  military  technical  information  transfer  methods 
should  be  supported  and  funded  as  a  three-  year  experiment. 

This  effort  will  analyze  information  presentation  approaches 
in  terras  of  their  application  to  commodity  groups  -  aircraft, 
combat  vehicles,  etc'.  -  and  will  select  the  best  approach  or 
combination  of  approaches  for  each  commodity  group.  Packages 
will  then  be  demonstrated  for  several  military  occupational 
specialties  for  maintenance  up  through  the  depot  level ., 
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C3)  The  Naval  Air  Systems  Command  ’  s  MIARS  and  TRUMP 
systems  should  be  accelerated  end  considered  for  Navy-wide 
use.  MIARS  is  a  microfilm  system  with  hard  copy  compatibility 
that  involves  the  reorganization  of  technical  data  into  "work 
packages"  that  cut  down  the  time  that  technicians  must  spend 
searching  through  manuals.  It  offers  the  capability  of  being 
the  vehicle  for  introducing  improvements  in  content  such  as  job 
performance  aids,  once  these  improvements  are  accepted.  TRUMP 
is  a  system  now  in  procurement  for  automatically  updating  'MIARS . 

Management: 

(1)  Focal  points  should  be  designated  in  OSD  and  in  each 
of  the  military  departments  with  responsibility  to  deal  with 
the  R&D,  systems,  logistics,  and  training  efforts  needed  for 
the  large  scale  demonstration  program. 

(2)  There  should  be  fiscal  visibility  and  management 
focus  for  the  R&D  program  for  maintenance  aids.  A  Program 
Element  should  be  established  for  each  military  department  to 
consolidate  the  efforts.. 

(3)  In  order  to  coordinate  the  overall  DOD  efforts, 
there  should  be  a  tri-service  review  of  current  activities 
and  plans  in  the  maintenance  aid  area.  A  committee  should  be 
formed  or  an  existing  one  directed  to  review  the  ongoing 
activity,  particularly  the  planning  of  major  demonstrations 
and  the  utilisation  of  the  results  of  those  demonstrations. 
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Formulation  of  an  ARPA  effort  similar  to  the  CAI  program 

0 

initiative  should  be  considered. 

(A)  For  an  interim  period  of  two  to  three  years,  OSD, 
specifically  OSD  (l&L)  and  DDRSE,  should  review  the  maintenance 
aids  approach  and  funding  proposed  for  new  weapon  systems 
and  should  stimulate  the  application  of  innovative  approaches 
and  funding  support  for  these. 

(5)  The  military  departments  should  initiate  analyses 
on  the  economics  of  production  and  utilization  of  improved, 
maintenance  aids.  These  should  include  the  comparative  costs 
of  alternative  format  and  content  (®.g.,  JPA's,  SIMMS,  con¬ 
ventional  manuals)  as  well  as  media  (books,  microfilm,  video 
cassette) . 

(6)  DOD  should  institute,  through  the  military  depart¬ 
ments,  an  analysis  of  the  costs  of  improper  maintenance  (to 
include  such  factors  as  fake  removals,  spares,  shipping,  etc.). 
This  data  should  be  included  in  a  broad  analysis  of  the 

economics  of  applying  better  maintenance  aids  and  information 

\  * 

systems  -  to  include  training  costs ,  personnel  costs ,  and 
equipment  costs. 

Systems  Application: 

(1)  Non-troubleshooting  fully  proceduralized  job  per¬ 
formance  aids  ■—  job  guides  —  should  be  developed  for  all 
new  weapons  systems.  These  will  be  primarily  oriented  to 
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routine  maintenance  task?.  This  should  be  done  in  accordance 
with  specifications  requiring  task  analysis. 

(2)  Job  guides  should  be  developed  for  selected  portions 
of  systems  now  in  the  inventory ,  based  on  a  review  of  high 
cost  areas. 

(3)  Information  systems  other  than  books  such  as  audio¬ 
visual  cassette  and  film  systems  to  provide  system  overviews , 
to  introduce  equipment  modifications  and  to  explain  difficult 
procedures  should  be  applied. 

(4)  For  major  weapon  systems,  microfilm  reader  printers 
which  include  job  guide  formatted  materials  should  be  used 
both  for  economic  update  and  facility  of  information 

access . 

Research  and  Development* 

(1)  FEFI/TAFX  (Flight  Environment  Fault  Indication/ 

Turn  Around  Fault  Isolation) ,  a  troubleshooting  system  for 
the  DC-10,  should  be  examined  for  possible  military  appli¬ 
cation  . 

(2)  Aids  useful  at  different  skill  and  experience  levels 
and  aids  that  have  training  objectives  should  be  developed. 

The  promising  Navy  sponsored  research  on  ’’families  of  aids” 
should  be  extended  so  that  multipurpose  documentation 
suitable  for  experienced  as  well  as  inexperienced  tech¬ 
nicians  can  be  developed  with  consideration  also  given 

to  the  inclusion  of  OJT  material. 
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(3)  Computer  supported  audio-visual  presentation 
systems  should  be  studied  as  potential  sophisticated  trouble¬ 
shooting  aids  making  possible  dynamically  optimized  trouble¬ 
shooting  strategies. 

(4)  Studies  should  be  made  of  the  influence  that  system 
characteristics  such  as  modularity  or  degree  of  built-in 
testing  have  on  the  type  of  aids  appropriate  for  a  system. 
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